[apiserver] Start apiserver v2 in apiserver/cmd/main.go #3603

troychiu · 2025-05-14T08:32:13Z

Why are these changes needed?

This PR start apiserver v2 in apiserver/cmd/main.go to ship the apiserver v2.

Related issue number

Closes #3596

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Signed-off-by: Troy Chiu <[email protected]>

troychiu · 2025-05-14T17:21:57Z

@rueian could you take a look? Thank you

apiserver/cmd/main.go

Signed-off-by: Troy Chiu <[email protected]>

rueian

LGTM

troychiu · 2025-05-14T22:07:58Z

@kevin85421 PTAL thank you!

apiserver/go.mod

apiserver/cmd/main.go

Signed-off-by: Troy Chiu <[email protected]>

apiserversdk/go.mod

kevin85421

I chatted with @rueian yesterday. Our conclusions are:

v1.4.0 is the last release where we fix bugs and add new features for v1.
We want to launch both v1 and v2 at the same time so that users can incrementally migrate from v1 to v2, and don't need to be 100% one time.
We don't want to create a new quay repository such as kuberay/apiserverv2. This may increase the complexity of migration.
For v1.5.0, the image ``kuberay/apiserver` will be v2 only.

apiserver/go.mod

kevin85421 · 2025-05-16T16:56:09Z

apiserver/cmd/main.go

@@ -144,8 +146,19 @@ func startHttpProxy() {
 	registerHttpHandlerFromEndpoint(ctx, api.RegisterRayServeServiceHandlerFromEndpoint, "ServeService", runtimeMux)
 	registerHttpHandlerFromEndpoint(ctx, api.RegisterRayJobSubmissionServiceHandlerFromEndpoint, "RayJobSubmissionService", runtimeMux)

+	kubernetesConfig, err := config.GetConfig()


Can you also add a feature gate (maybe a env var) for users to disable v2?

I added a flag instead, since that has been a pattern in api server

apiserver/cmd/main.go

Signed-off-by: Troy Chiu <[email protected]>

troychiu · 2025-05-20T05:50:35Z

apiserver/go.mod

@@ -28,6 +25,8 @@ require (
 	github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
 	github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.3
 	github.com/onsi/gomega v1.37.0
+	github.com/ray-project/kuberay/apiserversdk v0.0.0-20250519054718-f3ebea713a2f


since apiserversdk hasn't been released. I use the master commit instead.

Does it mean that the apiserver image uses different commits for apiserver v1 and v2? If so, it may cause some very hard to debug issues.

Fortunately #3640 solve this!

apiserver/go.mod

Signed-off-by: Troy Chiu <[email protected]>

apiserver/deploy/base/insecure/apiserver.yaml

apiserver/deploy/base/secure/apiserver.yaml

Signed-off-by: Troy Chiu <[email protected]>

troychiu · 2025-05-20T22:49:07Z

cc @kevin85421

kevin85421 · 2025-05-21T06:26:21Z

apiserver/deploy/base/insecure/apiserver.yaml

@@ -133,6 +133,32 @@ rules:
  - get
  - list
 ---
+# apiserversdk requires the following permissions to be able to list and proxy services


do we need to update Helm chart's RBAC? If you plan to open a follow up PR for Helm chart, can you open an issue to track the progress?

Yes I think we should. Thank you for catching this. Let me create an issue.

Created #3648

kevin85421 · 2025-05-21T06:28:37Z

apiserver/go.mod

@@ -28,6 +25,8 @@ require (
 	github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
 	github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.3
 	github.com/onsi/gomega v1.37.0
+	github.com/ray-project/kuberay/apiserversdk v0.0.0-20250519054718-f3ebea713a2f


Does it mean that the apiserver image uses different commits for apiserver v1 and v2? If so, it may cause some very hard to debug issues.

kevin85421 · 2025-05-21T06:33:33Z

apiserver/test/e2e/apiserversdk/cluster_e2e_test.go

+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
+	"github.com/ray-project/kuberay/apiserver/test/e2e"


Can we avoid importing apiserver? The payoff will come when we start deprecating v1. I am OK to add a file to add some test utils which copy some functions from apiserver/test/e2e.

Yeah that's a good point. I copied some utilities to e2e/apiserversdk and removed the unnecessary parts.

Signed-off-by: Troy Chiu <[email protected]>

kevin85421 · 2025-05-21T16:58:53Z

cc @rueian would you mind reviewing another iteration? Thanks!

apiserver/test/e2e/apiserversdk/event_e2e_test.go

Signed-off-by: Troy Chiu <[email protected]>

kevin85421 · 2025-05-21T18:35:45Z

@troychiu please ping me when all CI tests pass.

troychiu · 2025-05-21T19:45:15Z

@kevin85421 they passed! Thank you!

…3603)

use apiserversdk

4d95645

Signed-off-by: Troy Chiu <[email protected]>

troychiu changed the title ~~use apiserversdk~~ [apiserver] use apiserversdk May 14, 2025

troychiu changed the title ~~[apiserver] use apiserversdk~~ [apiserver] Start apiserver v2 in apiserver/cmd/main.go May 14, 2025

troychiu marked this pull request as ready for review May 14, 2025 17:20

rueian reviewed May 14, 2025

View reviewed changes

apiserver/cmd/main.go Outdated Show resolved Hide resolved

code review

e43cd01

Signed-off-by: Troy Chiu <[email protected]>

troychiu requested a review from rueian May 14, 2025 18:18

rueian approved these changes May 14, 2025

View reviewed changes

kevin85421 reviewed May 14, 2025

View reviewed changes

apiserver/go.mod Outdated Show resolved Hide resolved

apiserver/cmd/main.go Outdated Show resolved Hide resolved

fix go.mod

316f317

Signed-off-by: Troy Chiu <[email protected]>

rueian reviewed May 15, 2025

View reviewed changes

apiserversdk/go.mod Outdated Show resolved Hide resolved

kevin85421 reviewed May 16, 2025

View reviewed changes

rueian mentioned this pull request May 16, 2025

[DOCS] KubeRay APIServer V2 document #3594

Open

4 tasks

troychiu added 3 commits May 18, 2025 22:25

add e2e test

4e56ed8

Signed-off-by: Troy Chiu <[email protected]>

go.mod

be260a3

Signed-off-by: Troy Chiu <[email protected]>

add flag --enable-api-server-v2

c3fc87c

Signed-off-by: Troy Chiu <[email protected]>

troychiu commented May 20, 2025

View reviewed changes

apiserver/go.mod Outdated Show resolved Hide resolved

lint

8f7dafe

Signed-off-by: Troy Chiu <[email protected]>

troychiu requested review from rueian and kevin85421 May 20, 2025 06:42

rueian reviewed May 20, 2025

View reviewed changes

code review

74d6ea1

Signed-off-by: Troy Chiu <[email protected]>

kevin85421 reviewed May 21, 2025

View reviewed changes

troychiu added 3 commits May 20, 2025 23:52

merge master

39f469f

Signed-off-by: Troy Chiu <[email protected]>

remove dependency to apiserver v1

7dcd8cc

Signed-off-by: Troy Chiu <[email protected]>

refine

c0ed20e

Signed-off-by: Troy Chiu <[email protected]>

restore not needed

27266a4

Signed-off-by: Troy Chiu <[email protected]>

troychiu mentioned this pull request May 21, 2025

Update apiserver helm chart RBAC after introducing apiserver v2 #3648

Closed

rueian reviewed May 21, 2025

View reviewed changes

apiserver/test/e2e/apiserversdk/event_e2e_test.go Outdated Show resolved Hide resolved

code review

35903f4

Signed-off-by: Troy Chiu <[email protected]>

rueian approved these changes May 21, 2025

View reviewed changes

kevin85421 approved these changes May 21, 2025

View reviewed changes

kevin85421 merged commit 2ef997b into ray-project:master May 21, 2025
24 checks passed

pawelpaszki pushed a commit to opendatahub-io/kuberay that referenced this pull request May 26, 2025

[apiserver] Start apiserver v2 in apiserver/cmd/main.go (ray-project#…

4eb09b5

…3603)

[apiserver] Start apiserver v2 in apiserver/cmd/main.go #3603

[apiserver] Start apiserver v2 in apiserver/cmd/main.go #3603

Uh oh!

Conversation

troychiu commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

troychiu commented May 14, 2025

Uh oh!

Uh oh!

rueian left a comment

Choose a reason for hiding this comment

Uh oh!

troychiu commented May 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kevin85421 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

troychiu commented May 20, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kevin85421 commented May 21, 2025

Uh oh!

Uh oh!

kevin85421 commented May 21, 2025

Uh oh!

troychiu commented May 21, 2025

Uh oh!

Uh oh!

Uh oh!

troychiu commented May 14, 2025 •

edited

Loading